1. Technical Field
The present invention relates to computing systems, and specifically to systems that use high speed serial packet-switching fabrics, such as the InfiniBand architecture and PCI-Express.
2. Discussion of the Related Art
In current-generation computers, the central processing unit (CPU) is connected to the system memory and to peripheral devices by a shared parallel bus, such as the Peripheral Component Interface (PCI) bus or the Industry Standard Architecture (ISA) bus. Essentially, a bus is the channel or path between components in a computer. Likewise, current server to server connections and links to other server-related systems, such as remote storage and networking devices, depend on parallel bus technology. Server design dependent on a shared bus input/output (I/O) architecture may deliver for example 512 megabytes per second (MB/sec) of potential bandwidth that is shared among devices connected to the bus.
As data path-widths grow, and clock speeds become faster, the shared parallel bus becomes too costly and complex to keep up with system demands. In response, the computer industry is working to develop next-generation bus standards. Many of the proposed new standards have something in common. They propose doing away with the shared-bus technology used in PCI and moving to a point-to-point switching connection. Thus, the computer industry is moving towards fast, packetized, serial input/output bus architectures, in which computing hosts and peripherals are linked by a switching network, commonly referred to as a switching fabric. A number of architectures of this type have been proposed, and the first next-generation standard is already in place. InfiniBand has been advanced by a consortium headed by a group of industry leaders and PCI-Express is soon to follow.
The InfiniBand architecture is an I/O infrastructure technology that simplifies and speeds server-to-server connections and links to other server related systems, such as remote storage and network devices. The Infiniband fabric or mesh is the central network of connections between nodes, i.e., servers, and remote networking and storage devices within a data center. Infiniband architecture also constructs highly available data centers through multiple levels of redundancy. By connecting nodes via multiple links, Infiniband systems continue to perform even if one link fails. For increased reliability, multiple switches in a fabric provide redundant paths that allow for seamless re-routing of data through the fabric should a link between switches fail. Completely redundant fabrics may be configured for the highest level of reliability and may continue to perform even if an entire fabric fails.
Each node connects to the fabric through a channel adapter. Infiniband classifies the channel adapter into two categories: Host Channel Adapters (HCA) and Target Channel Adapters (TCA). HCAs are present in servers or even desktop machines and provide an interface that is used to integrate the InfiniBand with the operating system. TCAs are present on I/O devices such as a Redundant Array of Independent Disks (RAID) subsystem or a Just a Bunch of Disks (JBOD) subsystem. Each channel adapter may have one or more ports. A channel adapter with more than one port, may be connected to multiple switch ports. This allows for multiple paths between a source and a destination, resulting in performance and reliability benefits.
Infiniband channel adapters essentially have their own intelligence, therefore, they offload some of the communication processing from the operating system and CPU. Host and Target Channel adapters present an interface to the Open Systems Interconnection (OSI) reference model layers above them that allow those layers to generate and consume packets. In the case of a server writing a file to a storage device, the host is generating the packets that are then consumed by the storage device. In contrast to the channel adapter, switches simply forward packets between two of their ports based on an established routing table and the addressing information stored on the packets.
Infiniband uses Internet Protocol Version 6 (IPv6) headers natively, and can connect to Local Area Network (LAN) and Wide Area Network (WAN) switches and routers with the TCA providing a seamless transition between the system area network and external networks. InfiniBand defines network layers up to the transport layer and offers all four combinations of reliable/unreliable and connection datagram service. The Infiniband transport protocol is used within the system area network, but other transport protocols can be accessed by sending raw packets via a TCA. TCAs provide connections to storage, fibre channel networks, and other I/O nodes, and include an I/O controller specific to the device's protocol, be it Small Computer Systems Interface (SCSI), Ethernet, etc. A TCA includes an Infiniband protocol engine that dramatically accelerates the performance of critical Infiniband transport functions in the TCA's hardware, achieving aggregate internal transaction switching throughput speeds of 150 gigabits per second. TCAs are highly optimized for Infiniband target applications such as bridging from Infiniband devices (switches) to local busses i.e., Gigabit Ethernet, Fibre Channel, and Transport Control Protocol/Internet Protocol TCP/IP devices, as well as next-generation I/O protocols.